Classifier Calibration: with application to threat scores in cybersecurity

This paper explores the calibration of a classifier output score in binary classification problems. A calibrator is function that maps arbitrary score, testing observation, onto $[0,1]$ to provide an estimate for posterior probability belonging one two classes. Calibration important reasons; first, it provides meaningful probability; second, puts scores different classifiers on same scale comparable interpretation. The presents three main contributions: (1) Introducing multi-score calibration, when more than single observation. (2) idea process are nothing but features classifier, hence proposing expanding higher dimensions boost calibrator's performance. (3) Conducting massive simulation study, order 24,000 experiments, incorporates configurations, addition experimenting real datasets from cybersecurity domain. results show there no overall winner among calibrators and configurations. However, general advices practitioners include following: Platt's calibrator~\citep{Platt1999ProbabilisticOutputsForSupport}, version logistic regression decreases bias small sample size, has very stable acceptable performance all experiments; our suggested better majority including datasets. In addition, can help some experiments.